{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text.all import *\n",
    "from nbdev.showdoc import show_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ULMFiT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetune a pretrained Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we get our data and tokenize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts = get_files(path, extensions=['.txt'], folders=['unsup', 'train', 'test'])\n",
    "len(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we put it in a `Datasets`. For a language model, we don't have targets, so there is only one transform to numericalize the texts. Note that `tokenize_df` returns the count of the words in the corpus to make it easy to create a vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_file(f): return L(f.read().split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = RandomSplitter(valid_pct=0.1)(texts)\n",
    "tfms = [Tokenizer.from_folder(path), Numericalize()]\n",
    "dsets = Datasets(texts, [tfms], splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use that `Datasets` to create a `DataLoaders`. Here the class of `TfmdDL` we need to use is `LMDataLoader` which will concatenate all the texts in a source (with a shuffle at each epoch for the training set), split it in `bs` chunks then read continuously through it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,sl=256,80\n",
    "dbunch_lm = dsets.dataloaders(bs=bs, seq_len=sl, val_bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos i saw this before ' bubba ho - tep ' at the fantasia film festival in montreal . everything about it is either tipping the hat to ( or completely ripping off ) tim burton . i enjoyed it nonetheless , even if it is extremely derivative . what most impressed me was the quality of the visuals given the obvious shoe - string budget . the set design and the props were inventive and original , although the</td>\n",
       "      <td>i saw this before ' bubba ho - tep ' at the fantasia film festival in montreal . everything about it is either tipping the hat to ( or completely ripping off ) tim burton . i enjoyed it nonetheless , even if it is extremely derivative . what most impressed me was the quality of the visuals given the obvious shoe - string budget . the set design and the props were inventive and original , although the script</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>climax 25 minutes of sloppy wrap - up with a character and her dad that we do n't give a crap about anyway … xxunk line , xxup save xxup yourself … xxunk xxup from xxup this xxup movie xxrep 3 ! xxbos i never get tired of watching this movie . i am a die - hard chick - flick fan : fluff all the way , all that meaningless dime - a - dozen stuff . xxmaj but</td>\n",
       "      <td>25 minutes of sloppy wrap - up with a character and her dad that we do n't give a crap about anyway … xxunk line , xxup save xxup yourself … xxunk xxup from xxup this xxup movie xxrep 3 ! xxbos i never get tired of watching this movie . i am a die - hard chick - flick fan : fluff all the way , all that meaningless dime - a - dozen stuff . xxmaj but this</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>not hard for me to imagine a dedicated xxmaj mr . xxmaj xxunk teaching kids in xxmaj sunday xxmaj school about the good book . xxmaj nor is it hard to understand why they might picture xxunk 's guards in double - breasted suits like the gangsters in the news of their youth , or relating any number of other scenes to what was familiar to them . \\n\\n xxmaj connelly was not trying to convert viewers to religion …</td>\n",
       "      <td>hard for me to imagine a dedicated xxmaj mr . xxmaj xxunk teaching kids in xxmaj sunday xxmaj school about the good book . xxmaj nor is it hard to understand why they might picture xxunk 's guards in double - breasted suits like the gangsters in the news of their youth , or relating any number of other scenes to what was familiar to them . \\n\\n xxmaj connelly was not trying to convert viewers to religion … he</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>get better , but it never did . xxmaj from the start to the end , it was one big cliché , extremely predictable with not one surprise in the entire film . xxmaj from the over the top ridiculous boyfriend of xxmaj dawn and the wedding in the pie shop , the wife of the doctor being in the delivery room . i even found the scene where the husband finds the money and wants her to tell him</td>\n",
       "      <td>better , but it never did . xxmaj from the start to the end , it was one big cliché , extremely predictable with not one surprise in the entire film . xxmaj from the over the top ridiculous boyfriend of xxmaj dawn and the wedding in the pie shop , the wife of the doctor being in the delivery room . i even found the scene where the husband finds the money and wants her to tell him it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>up and sings a happy tune , but his mother comes in and tells him to shut up again and gives him a dope slap that leaves a dent in his forehead . i mention this commercial , because it was considered funny , and i did n't hear any objections to it while i was there . xxmaj there is a lot more bloodshed and physical cruelty on screen in \" the xxmaj great xxmaj yokai xxmaj war \"</td>\n",
       "      <td>and sings a happy tune , but his mother comes in and tells him to shut up again and gives him a dope slap that leaves a dent in his forehead . i mention this commercial , because it was considered funny , and i did n't hear any objections to it while i was there . xxmaj there is a lot more bloodshed and physical cruelty on screen in \" the xxmaj great xxmaj yokai xxmaj war \" than</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>this film portray our countries xxmaj special xxmaj forces . xxmaj gomer xxmaj pile could have probably survived longer than the \" spec xxmaj ops \" soldiers in this film . xxmaj for crying out loud they should have called them the xxmaj special xxmaj education xxmaj forces instead . xxmaj if you are going to write a script where you send in an elite team to deal with an outbreak of zombies , at least have the soldiers be</td>\n",
       "      <td>film portray our countries xxmaj special xxmaj forces . xxmaj gomer xxmaj pile could have probably survived longer than the \" spec xxmaj ops \" soldiers in this film . xxmaj for crying out loud they should have called them the xxmaj special xxmaj education xxmaj forces instead . xxmaj if you are going to write a script where you send in an elite team to deal with an outbreak of zombies , at least have the soldiers be smarter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>of high school ( they have to show at least 10 doors in the high school labeled \" debate xxmaj club , \" \" german xxmaj club , \" etc . ) and they tend to make fun of things like teen pregnancy and teen sex , which really has xxup nothing to do with making fun of horror films . xxmaj to say the least , i probably laughed once or twice through the entire 90 minutes , and</td>\n",
       "      <td>high school ( they have to show at least 10 doors in the high school labeled \" debate xxmaj club , \" \" german xxmaj club , \" etc . ) and they tend to make fun of things like teen pregnancy and teen sex , which really has xxup nothing to do with making fun of horror films . xxmaj to say the least , i probably laughed once or twice through the entire 90 minutes , and that</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>the xxmaj secret xxmaj xxunk . \\n\\n xxmaj however , xxmaj i 'm a little perplexed about how people have perceived her diary and of her as a person , seeing her as a little saint or having a message of hope for the world . i do n't think that was the original intention of her diary . xxmaj she wrote it mainly for herself , even though she did make some rigorous rewrites before the occupants of the</td>\n",
       "      <td>xxmaj secret xxmaj xxunk . \\n\\n xxmaj however , xxmaj i 'm a little perplexed about how people have perceived her diary and of her as a person , seeing her as a little saint or having a message of hope for the world . i do n't think that was the original intention of her diary . xxmaj she wrote it mainly for herself , even though she did make some rigorous rewrites before the occupants of the xxmaj</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>lied . xxmaj other facts are brought to light that , finally , result in xxmaj dillon 's release . xxmaj the killer is never found , though the movie gives us a thorough xxunk as a plausible perp . \\n\\n xxmaj this is a weeper from beginning to end . xxmaj nothing seems to go right for the couple . xxmaj oh , there are a few happy moment , maybe a party where everyone is glad to be</td>\n",
       "      <td>. xxmaj other facts are brought to light that , finally , result in xxmaj dillon 's release . xxmaj the killer is never found , though the movie gives us a thorough xxunk as a plausible perp . \\n\\n xxmaj this is a weeper from beginning to end . xxmaj nothing seems to go right for the couple . xxmaj oh , there are a few happy moment , maybe a party where everyone is glad to be together</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dbunch_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we have a convenience method to directly grab a `Learner` from it, using the `AWD_LSTM` architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Adam, wd=0.1)\n",
    "learn = language_model_learner(dbunch_lm, AWD_LSTM, opt_func=opt_func, metrics=[accuracy, Perplexity()], path=path)\n",
    "learn = learn.to_fp16(clip=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.426135</td>\n",
       "      <td>3.984901</td>\n",
       "      <td>0.292371</td>\n",
       "      <td>53.779987</td>\n",
       "      <td>07:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7,0.8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('stage1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('stage1');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.163227</td>\n",
       "      <td>3.870354</td>\n",
       "      <td>0.306840</td>\n",
       "      <td>47.959347</td>\n",
       "      <td>07:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.055693</td>\n",
       "      <td>3.790802</td>\n",
       "      <td>0.316436</td>\n",
       "      <td>44.291908</td>\n",
       "      <td>07:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.979279</td>\n",
       "      <td>3.729021</td>\n",
       "      <td>0.323357</td>\n",
       "      <td>41.638317</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.919654</td>\n",
       "      <td>3.688891</td>\n",
       "      <td>0.327770</td>\n",
       "      <td>40.000469</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.889432</td>\n",
       "      <td>3.660633</td>\n",
       "      <td>0.330762</td>\n",
       "      <td>38.885933</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.842923</td>\n",
       "      <td>3.637397</td>\n",
       "      <td>0.333315</td>\n",
       "      <td>37.992798</td>\n",
       "      <td>07:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.813823</td>\n",
       "      <td>3.619074</td>\n",
       "      <td>0.335308</td>\n",
       "      <td>37.303013</td>\n",
       "      <td>07:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.793213</td>\n",
       "      <td>3.608010</td>\n",
       "      <td>0.336566</td>\n",
       "      <td>36.892574</td>\n",
       "      <td>07:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.766456</td>\n",
       "      <td>3.602140</td>\n",
       "      <td>0.337257</td>\n",
       "      <td>36.676647</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.759768</td>\n",
       "      <td>3.600955</td>\n",
       "      <td>0.337450</td>\n",
       "      <td>36.633202</td>\n",
       "      <td>07:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(10, 2e-3, moms=(0.8,0.7,0.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have fine-tuned the pretrained language model to this corpus, we save the encoder since we will use it for the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('finetuned1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use it to train a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = get_files(path, extensions=['.txt'], folders=['train', 'test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = GrandparentSplitter(valid_name='test')(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For classification, we need to use two set of transforms: one to numericalize the texts and the other to encode the labels as categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_tfms = [Tokenizer.from_folder(path), Numericalize(vocab=dbunch_lm.vocab)]\n",
    "dsets = Datasets(texts, [x_tfms, [parent_label, Categorize()]], splits=splits, dl_type=SortedDL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = dsets.dataloaders(before_batch=pad_input_chunk, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos * * attention xxmaj spoilers * * \\n\\n xxmaj first of all , let me say that xxmaj rob xxmaj roy is one of the best films of the 90 's . xxmaj it was an amazing achievement for all those involved , especially the acting of xxmaj liam xxmaj neeson , xxmaj jessica xxmaj lange , xxmaj john xxmaj hurt , xxmaj brian xxmaj cox , and xxmaj tim xxmaj roth . xxmaj michael xxmaj canton xxmaj jones painted a wonderful portrait of the honor and dishonor that men can represent in themselves . xxmaj but alas … \\n\\n it constantly , and unfairly gets compared to \" braveheart \" . xxmaj these are two entirely different films , probably only similar in the fact that they are both about xxmaj scots in historical xxmaj scotland . xxmaj yet , this comparison frequently bothers me because it seems</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxbos xxmaj by now you 've probably heard a bit about the new xxmaj disney dub of xxmaj miyazaki 's classic film , xxmaj laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky . xxmaj during late summer of 1998 , xxmaj disney released \" kiki 's xxmaj delivery xxmaj service \" on video which included a preview of the xxmaj laputa dub saying it was due out in \" 1 xxrep 3 9 \" . xxmaj it 's obviously way past that year now , but the dub has been finally completed . xxmaj and it 's not \" laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky \" , just \" castle xxmaj in xxmaj the xxmaj sky \" for the dub , since xxmaj laputa is not such a nice word in xxmaj spanish ( even though they use the word xxmaj laputa many times</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we once again have a convenience function to create a classifier from this `DataLoaders` with the `AWD_LSTM` architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Adam, wd=0.1)\n",
    "learn = text_classifier_learner(dls, AWD_LSTM, metrics=[accuracy], path=path, drop_mult=0.5, opt_func=opt_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load our pretrained encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = learn.load_encoder('finetuned1')\n",
    "learn = learn.to_fp16(clip=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can train with gradual unfreezing and differential learning rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-1 * bs/128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.328318</td>\n",
       "      <td>0.200650</td>\n",
       "      <td>0.922120</td>\n",
       "      <td>01:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, lr, moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.208120</td>\n",
       "      <td>0.166004</td>\n",
       "      <td>0.937440</td>\n",
       "      <td>01:15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "lr /= 2\n",
    "learn.fit_one_cycle(1, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.162498</td>\n",
       "      <td>0.154959</td>\n",
       "      <td>0.942400</td>\n",
       "      <td>01:35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "lr /= 2\n",
    "learn.fit_one_cycle(1, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.133800</td>\n",
       "      <td>0.163456</td>\n",
       "      <td>0.940560</td>\n",
       "      <td>01:34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.095326</td>\n",
       "      <td>0.154301</td>\n",
       "      <td>0.945120</td>\n",
       "      <td>01:34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "lr /= 5\n",
    "learn.fit_one_cycle(2, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
